40 research outputs found
Performance Metrics for Probabilistic Ordinal Classifiers
Ordinal classification models assign higher penalties to predictions further
away from the true class. As a result, they are appropriate for relevant
diagnostic tasks like disease progression prediction or medical image grading.
The consensus for assessing their categorical predictions dictates the use of
distance-sensitive metrics like the Quadratic-Weighted Kappa score or the
Expected Cost. However, there has been little discussion regarding how to
measure performance of probabilistic predictions for ordinal classifiers. In
conventional classification, common measures for probabilistic predictions are
Proper Scoring Rules (PSR) like the Brier score, or Calibration Errors like the
ECE, yet these are not optimal choices for ordinal classification. A PSR named
Ranked Probability Score (RPS), widely popular in the forecasting field, is
more suitable for this task, but it has received no attention in the image
analysis community. This paper advocates the use of the RPS for image grading
tasks. In addition, we demonstrate a counter-intuitive and questionable
behavior of this score, and propose a simple fix for it. Comprehensive
experiments on four large-scale biomedical image grading problems over three
different datasets show that the RPS is a more suitable performance metric for
probabilistic ordinal predictions. Code to reproduce our experiments can be
found at https://github.com/agaldran/prob_ord_metrics .Comment: Accepted to MICCAI 202
Visibility Recovery on Images Acquired in Attenuating Media. Application to Underwater, Fog, and Mammographic Imaging
When acquired in attenuating media, digital images often suffer from a
particularly complex degradation that reduces their visual quality, hindering
their suitability for further computational applications, or simply
decreasing the visual pleasantness for the user. In these cases, mathematical
image processing reveals itself as an ideal tool to recover some
of the information lost during the degradation process. In this dissertation,
we deal with three of such practical scenarios in which this problematic
is specially relevant, namely, underwater image enhancement, fog
removal and mammographic image processing. In the case of digital mammograms,
X-ray beams traverse human tissue, and electronic detectors
capture them as they reach the other side. However, the superposition
on a bidimensional image of three-dimensional structures produces lowcontrasted
images in which structures of interest suffer from a diminished
visibility, obstructing diagnosis tasks. Regarding fog removal, the loss
of contrast is produced by the atmospheric conditions, and white colour
takes over the scene uniformly as distance increases, also reducing visibility.
For underwater images, there is an added difficulty, since colour is not
lost uniformly; instead, red colours decay the fastest, and green and blue
colours typically dominate the acquired images. To address all these challenges,
in this dissertation we develop new methodologies that rely on: a)
physical models of the observed degradation, and b) the calculus of variations.
Equipped with this powerful machinery, we design novel theoretical
and computational tools, including image-dependent functional energies
that capture the particularities of each degradation model. These energies
are composed of different integral terms that are simultaneously
minimized by means of efficient numerical schemes, producing a clean,
visually-pleasant and useful output image, with better contrast and increased
visibility. In every considered application, we provide comprehensive
qualitative (visual) and quantitative experimental results to validate
our methods, confirming that the developed techniques outperform other
existing approaches in the literature
The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
In spite of the dominant performances of deep neural networks, recent works
have shown that they are poorly calibrated, resulting in over-confident
predictions. Miscalibration can be exacerbated by overfitting due to the
minimization of the cross-entropy during training, as it promotes the predicted
softmax probabilities to match the one-hot label assignments. This yields a
pre-softmax activation of the correct class that is significantly larger than
the remaining activations. Recent evidence from the literature suggests that
loss functions that embed implicit or explicit maximization of the entropy of
predictions yield state-of-the-art calibration performances. We provide a
unifying constrained-optimization perspective of current state-of-the-art
calibration losses. Specifically, these losses could be viewed as
approximations of a linear penalty (or a Lagrangian) imposing equality
constraints on logit distances. This points to an important limitation of such
underlying equality constraints, whose ensuing gradients constantly push
towards a non-informative solution, which might prevent from reaching the best
compromise between the discriminative performance and calibration of the model
during gradient-based optimization. Following our observations, we propose a
simple and flexible generalization based on inequality constraints, which
imposes a controllable margin on logit distances. Comprehensive experiments on
a variety of image classification, semantic segmentation and NLP benchmarks
demonstrate that our method sets novel state-of-the-art results on these tasks
in terms of network calibration, without affecting the discriminative
performance. The code is available at https://github.com/by-liu/MbLS .Comment: To Appear at CVPR 2022. Code: https://github.com/by-liu/MbL
Class Adaptive Network Calibration
Recent studies have revealed that, beyond conventional accuracy, calibration
should also be considered for training modern deep neural networks. To address
miscalibration during learning, some methods have explored different penalty
functions as part of the learning objective, alongside a standard
classification loss, with a hyper-parameter controlling the relative
contribution of each term. Nevertheless, these methods share two major
drawbacks: 1) the scalar balancing weight is the same for all classes,
hindering the ability to address different intrinsic difficulties or imbalance
among classes; and 2) the balancing weight is usually fixed without an adaptive
strategy, which may prevent from reaching the best compromise between accuracy
and calibration, and requires hyper-parameter search for each application. We
propose Class Adaptive Label Smoothing (CALS) for calibrating deep networks,
which allows to learn class-wise multipliers during training, yielding a
powerful alternative to common label smoothing penalties. Our method builds on
a general Augmented Lagrangian approach, a well-established technique in
constrained optimization, but we introduce several modifications to tailor it
for large-scale, class-adaptive training. Comprehensive evaluation and multiple
comparisons on a variety of benchmarks, including standard and long-tailed
image classification, semantic segmentation, and text classification,
demonstrate the superiority of the proposed method. The code is available at
https://github.com/by-liu/CALS.Comment: Code: https://github.com/by-liu/CAL
Calibrating Segmentation Networks with Margin-based Label Smoothing
Despite the undeniable progress in visual recognition tasks fueled by deep
neural networks, there exists recent evidence showing that these models are
poorly calibrated, resulting in over-confident predictions. The standard
practices of minimizing the cross entropy loss during training promote the
predicted softmax probabilities to match the one-hot label assignments.
Nevertheless, this yields a pre-softmax activation of the correct class that is
significantly larger than the remaining activations, which exacerbates the
miscalibration problem. Recent observations from the classification literature
suggest that loss functions that embed implicit or explicit maximization of the
entropy of predictions yield state-of-the-art calibration performances. Despite
these findings, the impact of these losses in the relevant task of calibrating
medical image segmentation networks remains unexplored. In this work, we
provide a unifying constrained-optimization perspective of current
state-of-the-art calibration losses. Specifically, these losses could be viewed
as approximations of a linear penalty (or a Lagrangian term) imposing equality
constraints on logit distances. This points to an important limitation of such
underlying equality constraints, whose ensuing gradients constantly push
towards a non-informative solution, which might prevent from reaching the best
compromise between the discriminative performance and calibration of the model
during gradient-based optimization. Following our observations, we propose a
simple and flexible generalization based on inequality constraints, which
imposes a controllable margin on logit distances. Comprehensive experiments on
a variety of public medical image segmentation benchmarks demonstrate that our
method sets novel state-of-the-art results on these tasks in terms of network
calibration, whereas the discriminative performance is also improved.Comment: Under review. The code is available at
https://github.com/Bala93/MarginLoss. arXiv admin note: substantial text
overlap with arXiv:2111.1543
The Little W-Net That Could: State-of-the-Art Retinal Vessel Segmentation with Minimalistic Models
The segmentation of the retinal vasculature from eye fundus images represents
one of the most fundamental tasks in retinal image analysis. Over recent years,
increasingly complex approaches based on sophisticated Convolutional Neural
Network architectures have been slowly pushing performance on well-established
benchmark datasets. In this paper, we take a step back and analyze the real
need of such complexity. Specifically, we demonstrate that a minimalistic
version of a standard U-Net with several orders of magnitude less parameters,
carefully trained and rigorously evaluated, closely approximates the
performance of current best techniques. In addition, we propose a simple
extension, dubbed W-Net, which reaches outstanding performance on several
popular datasets, still using orders of magnitude less learnable weights than
any previously published approach. Furthermore, we provide the most
comprehensive cross-dataset performance analysis to date, involving up to 10
different databases. Our analysis demonstrates that the retinal vessel
segmentation problem is far from solved when considering test images that
differ substantially from the training data, and that this task represents an
ideal scenario for the exploration of domain adaptation techniques. In this
context, we experiment with a simple self-labeling strategy that allows us to
moderately enhance cross-dataset performance, indicating that there is still
much room for improvement in this area. Finally, we also test our approach on
the Artery/Vein segmentation problem, where we again achieve results
well-aligned with the state-of-the-art, at a fraction of the model complexity
in recent literature. All the code to reproduce the results in this paper is
released
Automatic Red-Channel underwater image restoration.
Underwater images typically exhibit color distortion and low contrast as a result of the exponential decay that light suffers as it travels. Moreover, colors associated to different wavelengths have different attenuation rates, being the red wavelength the one that attenuates the fastest. To restore underwater images, we propose a Red Channel method, where colors associated to short wavelengths are recovered, as expected for underwater images, leading to a recovery of the lost contrast. The Red Channel method can be interpreted as a variant of the Dark Channel method used for
images degraded by the atmosphere when exposed to haze. Experimental results show that our technique handles gracefully artificially illuminated areas, and achieves a natural color correction and superior or equivalent visibility improvement when compared to other state-of-the-art methods